Information processing apparatus, operation method thereof, and computer program

ABSTRACT

An information processing apparatus includes: an image acquiring unit configured to acquire a plurality of types of images of an eye, including a confocal image and a non-confocal image of the eye; an analyzing unit configured to analyze the confocal image and the non-confocal image; a deciding unit configured to decide, based on analysis results of one of the confocal image and the non-confocal image, whether or not to analyze the other; and a display control unit configured to display analysis results of the confocal image and the non-confocal image on a display unit, in a case of deciding to analyze the other.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an information processing apparatusused in ophthalmological diagnosis and treatment, an operation methodthereof, and a computer program.

2. Description of the Related Art

Examination of the eye is widely performed for early diagnosis andtreatment of lifestyle diseases and diseases which are primary causes ofloss of eyesight. The scanning laser ophthalmoscope (SLO), which is anophthalmological apparatus that employs the principle of confocal laserscanning microscopy, performs high speed raster scanning of a subject'seye with a laser beam, which is measurement light, and acquires ahigh-resolution planar image of the fundus from the intensity ofreturning light.

In confocal laser scanning microscopy, detecting only light that haspassed through an aperture (pinhole) enables an image to be formed justusing returning light of a particular depth position (focal point), andtherefore images with higher contrast than those obtained by funduscameras and the like can be acquired. An apparatus that obtains suchhigh-contrast planar images will hereinafter be referred to as an SLOapparatus, and a planar image obtained thusly is referred to as an SLOimage.

In recent years, increased beam diameter of measurement light in SLOapparatuses has enabled acquisition of SLO images of the retina, withimproved horizontal resolution. However, the increased beam diameter ofthe measurement light has led to deterioration of the S/N ratio and ofthe resolution of the SLO image during acquisition of SLO images of theretina, due to aberration of the eye being examined. An adaptive opticsSLO apparatus has been developed to counter the deterioration of S/Nratio and improve resolution of the SLO image. The adaptive optics SLOapparatus has an adaptive optics system which includes a wavefrontsensor and a wavefront correction device. The wave front sensor measuresin real time wavefront aberrations caused by the eye being examined, andthe wavefront correction device corrects the wavefront aberration withregard to the measurement light and the returning light. This enablesthe acquisition of SLO images with high resolution in the horizontal ormain-scanning direction so that a high-magnification image can beacquired.

Such a high resolution SLO image can be acquired as a moving image. Inorder to noninvasively observe the dynamics of blood flow(hemodynamics), for example, retinal blood vessels are extracted fromeach frame of an SLO image, and the moving speed of blood cells throughcapillaries and so forth is measured by performing image analysis. Also,in order to evaluate the visual function of an eye using an SLO image,photoreceptors P are detected, and the density distribution andarrangement (array) of the photoreceptors P are calculated.

However, confocal images taken of the inner layers of the retina haveintense noise signals due to the influence of light reflecting from thenerve fiber layer, and there have been cases where observing bloodvessel walls and detection of wall boundaries has been difficult.Accordingly, as of recent, techniques have come into use for observationof non-confocal images obtained by acquiring scattered light, bychanging the diameter, shape, and position of a pinhole on the near sideof the light receiving portion. An example of this technique isdescribed in Sulai, Dubra et al.; “Visualization of retinal vascularstructure and perfusion with a nonconfocal adaptive optics scanninglight ophthalmoscope”, J. Opt. Soc. Am. A, Vol. 31, No. 3, pp. 569-579,2014 (hereinafter “Sulai and Dubra”). Non-confocal images have a greatdepth of focus, so objects that have unevenness in the depth direction,such as blood vessels can be easily observed, and also noise is reducedsince reflected light from the nerve fiber layer is not readily directlyreceived. While observation of photoreceptors at the outer layers of theretina has primarily involved imaging confocal images of the outersegment of photoreceptors, it has been found that the unevenness of theinner segment of photoreceptors can be imaged in non-confocal images.This is described in Scoles, Dubra et al.; “In vivo Imaging of HumanCone Photoreceptor Inner Segment”, IOVS, Vol. 55, No. 7, pp. 4244-4251,2014 (hereinafter “Scoles and Dubra”). Sulai and Dubra disclosetechnology for acquiring non-confocal images of retinal blood vesselsusing an adaptive optics SLO apparatus, while Scoles and Dubra disclosetechnology for acquiring both confocal images and non-confocal images atthe same time using an adaptive optics SLO apparatus. However, theseknown techniques lack a method of efficiently processing and analyzingconfocal images and non-confocal images to accurately determine whetherthe confocal image and the non-confocal image yields better imagingresults.

SUMMARY OF THE INVENTION

Embodiments of an information processing apparatus according to thepresent invention and an operation method thereof have the followingconfigurations, for example. An information processing apparatusaccording to an aspect of the present invention includes: an imageacquiring unit configured to acquire a plurality of types of images ofan eye, including a confocal image and a non-confocal image of the eye;an analyzing unit configured to analyze the confocal image and thenon-confocal image; a deciding unit configured to decide, based onanalysis results of one of the confocal image and the non-confocalimage, whether or not to analyze the other; and a display control unitconfigured to display analysis results of the confocal image and thenon-confocal image on a display unit, in a case of deciding to analyzethe other.

An operation method of an information processing apparatus according toan aspect of the present invention includes: a step of acquiring aplurality of types of images of an eye, including a confocal image and anon-confocal image of the eye; a step of deciding, based on analysisresults of one of the confocal image and the non-confocal image, whetheror not to analyze the other; and a step of displaying analysis resultsof the confocal image and the non-confocal image on a display unit, in acase of deciding to analyze the other.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a functional configurationexample of an information processing apparatus according to a firstembodiment.

FIGS. 2A and 2B are block diagrams illustrating configuration examplesof a system including the information processing apparatus according tothe first embodiment.

FIGS. 3A through 3H are diagrams for describing the overallconfiguration of an SLO image imaging apparatus according to the firstembodiment.

FIG. 4 is a block diagram illustrating a hardware configuration exampleof a computer which has hardware equivalent to a storage unit and imageprocessing unit and holds other units as software which is executed.

FIG. 5 is a flowchart of processing which the information processingapparatus according to the first embodiment executes.

FIGS. 6A through 6M are diagrams illustrating what is performed in imageprocessing according to the first embodiment.

FIGS. 7A through 7C are flowcharts illustrating the details ofprocessing executed in S520, S530, and S540 according to the firstembodiment.

FIG. 8 is a block diagram illustrating a functional configurationexample of an information processing apparatus according to a secondembodiment.

FIGS. 9A through 9C are flowcharts illustrating the details ofprocessing executed in S520, S530, and S540 according to the secondembodiment.

FIG. 10 is a block diagram illustrating a functional configurationexample of an information processing apparatus according to a thirdembodiment.

FIGS. 11A through 11D are flowcharts illustrating the details ofprocessing executed in S520 and S530 according to the third embodimentand a fourth embodiment.

FIG. 12 is a block diagram illustrating a functional configurationexample of an information processing apparatus according to the fourthembodiment.

DESCRIPTION OF THE EMBODIMENTS

SLO apparatuses that acquire confocal images and non-confocal imagesgenerally read all images into memory, and perform inter-imagecalculation processing or image measurement processing. This isperformed in order to improve the reliability of analysis by analyzingimages of different types.

It has been found desirable to effectively analyze multiple types ofimages of the eye, including non-confocal images of the eye, whileimproving reliability of analysis.

Accordingly, one aspect of an embodiment includes an image acquiringunit configured to acquire multiple types of images of an eye includingat least one type of non-confocal image of the eye (as an example, adata acquiring unit 110 in FIG. 1). One aspect of an embodiment includesa deciding unit 131 configured to decide an analyzing method to analyzethe multiple types of images that are acquired (as an example, adeciding unit 131 in FIG. 1). One aspect of an embodiment includes ananalyzing unit 130 (as an example, an image processing unit 130 inFIG. 1) configured to analyze at least one of the multiple types ofimages based on the decided analysis method. Accordingly, reliability ofanalysis can be improved while efficiently analyzing multiple types ofimages of the eye including at least one type of non-confocal image ofthe eye.

Another aspect of an embodiment includes an image acquiring unitconfigured to acquire multiple types of images including a confocalimage and at least one type of non-confocal image of an eye (as anexample, a data acquiring unit 110 in FIG. 1). Another aspect of anembodiment includes an analyzing unit configured to analyze a confocalimage and at least one type of non-confocal image (as an example, theimage processing unit 130 in FIG. 1). Another aspect of an embodimentincludes a deciding unit configured to decide whether or not to analyzethe at least one type of non-confocal image based on analysis results ofthe confocal image (as an example, the deciding unit 131 in FIG. 1).Another aspect of an embodiment includes a display control unitconfigured to display analysis results of a confocal image and at leastone type of non-confocal image on a display unit, in a case of decidingto analyze the at least one type of non-confocal image (as an example,the display control unit 133 in FIG. 1). Accordingly, reliability ofanalysis can be improved while efficiently analyzing multiple types ofimages of the eye including a confocal image and at least one type ofnon-confocal image of the eye. Note that the deciding unit may decidewhether or not to analyze the confocal image based on analysis resultsof the non-confocal image.

Another aspect of an embodiment preferably efficiently performs imageprocessing upon having decided the object and method of image processingin accordance with attributes of imaged images, whether the imagesinclude anatomical features or disorder portions, and the quality of theimage (image quality, and to what degree an imaging object region isincluded). That is to say, an apparatus that acquires multiple types ofimages with different light receiving methods performs image processingwith priority on images which are more important with regard toobservation and measurement necessitates technology of generating ormeasuring a great number of images more efficiently.

As noted above, the technology described in Sulai and Dubra disclosestechnology relating to an adaptive optics SLO apparatus that acquiresmulti-channel non-confocal images, bud does not disclose a method toefficiently generate or measure a great number of types of non-confocalimages. Although the technology described in Scoles and Dubra acquiresconfocal images and non-confocal images at the same time, there is nodisclosure of a method of efficiently generating or measuring confocalimages and non-confocal images.

Embodiments of an image processing apparatus, an operation methodthereof, and a computer program, according to the present invention,will be described below with reference to the attached drawings. Itshould be noted, though, that the present invention is not restricted tothis description.

First Embodiment Deciding Image Processing Method for Each Image TypeBeforehand

An information processing apparatus 10 according to the presentembodiment is configured to uniformly perform image computation andmeasurement according to a method specified for each image attribute(imaging position and image type) of captured images of photoreceptorswhich are an example of an observation object, as necessary inobservation and measurement. This is performed using an SLO apparatuswhich is an example of an ophthalmic imaging apparatus. Specifically, ofconfocal images Dc and non-confocal images Dn of photoreceptors P takenat the macular area, confocal images Dcj and Split Detector images Dnsktaken within 1.5 mm from the fovea, crucial in observation andmeasurement, are composited to generate composited images. In otherwords, images of which the center of photography is farther than 1.5 mmfrom the fovea, and R channel images Dnrk and L channel images Dn1 kwithin 1.5 mm from the fovea are not used in image generation of acomposite image, or in measurement of blood flow dynamics.

Overall Configuration

FIGS. 2A and 2B are configuration diagrams of a system including theinformation processing apparatus 10 according to the present embodiment.The information processing apparatus 10 is communicably connected to anSLO image imaging apparatus 20, which is an example of an ophthalmicimaging apparatus, a data server 40, and a time phase data acquisitionapparatus 50, via a local area network (LAN) 30 including optical fiber,Universal Serial Bus (USB), IEEE 1394, or the like, as illustrated inFIGS. 2A and 2B. The configuration of communicable connection to thesedevices may be via an external network such as the Internet, or may be aconfiguration where the information processing apparatus 10 is directlyconnected to the SLO image imaging apparatus 20. Alternatively, theinformation processing apparatus 10 may be integrally built into anophthalmic imaging apparatus.

The SLO image imaging apparatus 20 is an apparatus to image confocalimages Dc and non-confocal images Dn, which are wide-angle images D1 andhigh-magnification images Dh of the eye. The SLO image imaging apparatus20 transmits wide-angle images D1, confocal images Dc, non-confocalimages Dn, and information of fixation target positions F1 and Fcn usedfor imaging thereof, to the information processing apparatus 10 and thedata server 40. In a case where these images are acquired at differentimaging positions, this is expressed as D1 i, Dcj, Dnk. That is to say,i and j are variables indicating the numbers for the imaging positions,where i=1, 2, . . . , imax, j=1, 2, . . . , jmax and k=1, 2, . . . ,kmax. In a case of acquiring confocal images Dc and non-confocal imagesDn at different magnifications, this is expressed like Dc1 m, Dc2 o, . .. (Dn1 m, Dn2 o, . . . ) in order from the highest-magnification image,with Dc1 m (Dn1 m) denoting high-magnification confocal (non-confocal)images, and Dc2 o, . . . (Dn2 o, . . . ) denoting mid-magnificationimages.

The data server 40 holds the wide-angle images D1, confocal images Dc,and non-confocal images Dn, of the examinee eye, imaging conditions datasuch as fixation target positions F1 and Fcn used for the imagingthereof, image features of the eye, and so forth. In the presentinvention, image features relating to the photoreceptors P, capillariesQ, blood cells W, and retinal blood vessel walls, are handled as imagefeatures of the eye. The wide-angle images D1, confocal images Dc, andnon-confocal images Dn output from the SLO image imaging apparatus 20,fixation target positions F1 and Fcn used for the imaging thereof, andimage features of the eye output from the information processingapparatus 10, are saved in the server 40. Also, the wide-angle imagesD1, confocal images Dc, and non-confocal images Dn, and image featuresof the eye, are transmitted to the information processing apparatus 10in response to requests from the information processing apparatus 10.

FIG. 6B illustrates an example of a high resolution SLO image. In FIG.6B, the photoreceptors P, a low-luminance region Q corresponding to theposition of capillaries, and a high-luminance region W corresponding tothe position of a white blood cell, can be observed. In a case ofobserving photoreceptors P in the SLO image, the focus position is setnearby the outer layer of the retina (B5 in FIG. 6A) to take a SLO imagesuch as the one shown in FIG. 6B. On the other hand, there are retinalblood vessels and capillaries that have branched running through theinner layers of the retina (B2, B3 through B4 in FIG. 6A). Acquiring anSLO image with adaptive optics allows setting the focus position in theinner layers of the retina, and thus adaptive optics SLO imaging enablesthe retinal blood vessel walls to be directly observed.

Next, the functional configuration of the information processingapparatus 10 according to the present embodiment will be described withreference to FIG. 1. FIG. 1 is a block diagram illustrating thefunctional configuration of the information processing apparatus 10. Theinformation processing apparatus 10 includes a data acquiring unit 110,a storage unit 120, an image processing unit 130, and an instructionacquiring unit 140. The data acquiring unit 110 includes a confocal dataacquiring unit 111, a non-confocal data acquiring unit 112, and anattribute acquiring unit 113. The image processing unit 130 includes adeciding unit 131, a positioning unit 132, and a display control unit133. The deciding unit 131 further has a determining unit 1311 and animage processing method deciding unit 1312.

Now, the SLO imaging apparatus 20 that applies adaptive optics will bedescribed with reference to FIGS. 3A and 3B. The SLO imaging apparatus20 includes a superluminescent diode (SLD) 201, a Shack-Hartmanwavefront sensor 206, an adaptive optics system 204, beam splitters 202and 203, an X-Y scanning mirror 205, a focus lens 209, a diaphragm 210,a photosensor 211, an image forming unit 212, and an output unit 213.

Light irradiated from the SLD 201 that is the light source is reflectedat the fundus. Part of the reflected light is input to the Shack-Hartmanwavefront sensor 206 via the second beam splitter 203, and the remainingreflected light is input to the photosensor 211 via the first beamsplitter 202. Although the light source here services both as a lightsource for acquiring confocal images and a light source for acquiringnon-confocal images, multiple light sources configured to emit differentwavelengths may be used, or the like. The Shack-Hartman wavefront sensor206 is a device to measure aberration of the eye, in which a lens eye207 is connected to a charge-coupled device (CCD) 208. Upon input lightbeing transmitted through the lens eye 207, bright point set appears onthe CCD 208, and wave aberration is measured base on the positional gapof the projected bright points. The adaptive optics system 204 drives anaberration correction device (deformable mirror or spatial light phasemodulator) to correct the aberration, based on the wave aberrationmeasured by the Shack-Hartman wavefront sensor 206. The light subjectedto aberration-correction passes through the focus lens 209 and diaphragm210, and is received at the photosensor 211. The diaphragm 210 andphotosensor 211 are examples of the aperture and optical detectoraccording to the present invention. The aperture (of the diaphragm 210)preferably is provided upstream of and near to the optical detector(photosensor 211). The scanning position on the fundus can be controlledby moving the X-Y scanning mirror 205, thereby acquiring data accordingto an imaging region and time (frame rate×frame count) that the operatorhas instructed. The data is transmitted to the image forming unit 212,where image distortion due to variation in scanning rate is correctedand luminance value correction is performed, thereby forming image data(moving image or still image). The output unit 213 outputs the imagedata formed by the image forming unit 212.

The configuration of the diaphragm 210 and photosensor 211 portion inFIG. 3A is optional, just as long as the SLO imaging apparatus 20 isconfigured to be able to acquire confocal images Dc and non-confocalimages Dn. The present embodiment is configured using a light-shieldingmember 210-1 (FIGS. 3B and 3E) and photosensor 211-1, 211-2, and 211-3(FIG. 3B). Regarding the returning light in FIG. 3B, part of the lightthat has entered the light-shielding member 210-1 disposed at the imageforming plate is reflected and enters the photosensor 211-1. Now, thelight-shielding member 210-1 will be described with reference to FIG.3E. The light-shielding member 210-1 is made up of transmitting regions210-1-2 and 210-1-3, a light-shielded region (omitted fromillustration), and a reflecting region 210-1-1, so that the center ispositioned on the center of the optical axis of the returning light. Thelight-shielding member 210-1 has an elliptic shape pattern so that whendisposed obliquely as to the optical axis of the returning light, theshape appears to be circular when seen from the optical axis direction.The returning light divided at the light-shielding member 210-1 is inputto the photosensor 211-1. The returning light that has passed throughthe transmitting regions 210-1-2 and 210-1-3 of the light-shieldingmember 210-1 is split by a prism 210-2 disposed at the image formingplane, and is input to photosensors 211-2 and 211-3, as illustrated inFIG. 3B. Voltage signals obtained at the photosensors are converted intodigital values at an AD board within the image forming unit 212, therebyforming a two-dimensional image. The image based on light entering thephotosensor 211-1 is a confocal image where focus has been made on aparticular narrow range. Images based on light entering the photosensors211-2 and 211-3 are non-confocal images where focus has been made on abroad range. The light-shielding member 210-1 is an example of anoptical member that divides returning light from the eye which has beenirradiated by light from the light source, into returning light passingthrough a confocal region and returning light passing through anon-confocal region. The transmitting regions 210-1-2 and 210-1-3 areexamples of a non-confocal region, and non-confocal images are acquiredbased on the returning light passing through the non-confocal regions.The reflecting region 210-1-1 is an example of a confocal region, andconfocal images are acquired based on the returning light passingthrough the confocal region.

The method for dividing non-confocal signals is not restricted to this,and a configuration may be made where non-confocal signals are dividedinto four and received, such as illustrated in FIG. 3F, for example.Also, the reception method of confocal signals and non-confocal signalsis not restricted to this. For example, a mechanism is preferably hadwhere the diameter and position of the diaphragm 210 (aperture) ischangeable. In doing so, at least one of the diameter of the apertureand the position in the optical axis direction is configured so as to beadjustable, so as to receive as confocal signals as illustrated in FIG.3C and to receive as non-confocal signals as illustrated in FIG. 3D. Thediameter and movement amount of the aperture may be optionally adjusted.For example, FIG. 3C shows that the diameter of the aperture can beadjusted to around 1 Airy disc diameter (ADD), and FIG. 3D shows thatthe diameter of the aperture can be adjusted to around 10 ADD with amovement amount of around 6 ADD. Alternatively, a configuration may bemade where multiple non-confocal signals are received at the same time,as in FIGS. 3G and 3H. There are two types of non-confocal signals inthe present embodiment, so one will be denoted by Dnr referring to the Rchannel image, and the other will be denoted by Dn1 referring to the Lchannel image. The notation “non-confocal image Dn” refers to both the Rchannel image Dnr and L channel image Dn1.

The SLO imaging apparatus 20 can also operate as a normal SLO apparatus,by increasing the scan angle of the scanning optical system in theconfiguration in FIG. 3A, and instructing so that the adaptive opticssystem 204 does not perform aberration correction, so as to imagewide-angle confocal images and non-confocal images. Images which arelower magnification than the high-magnification images Dc and Dn, andhave the lowest magnification of images acquired by the data acquiringunit 110 will be referred to as wide-angle images D1 (D1 c, D1 r, D1 l).Accordingly, a wide-angle image D1 may be an SLO image where adaptiveoptics has been applied, and cases of a simple SLO image are alsoincluded. Note that when distinguishing between confocal wide-angleimages and non-confocal wide-angle images D1, these are denoted by D1 c,D1 r, and D1 l.

Next, the hardware configuration of the information processing apparatus10 will be described with reference to FIG. 4. In FIG. 4, 301 denotes acentral processing unit (CPU), 302 memory (random access memory (RAM)),303 control memory (read-only memory (ROM)), 304 an external storagedevice, 305 a monitor, 306 a keyboard, 307 a mouse, and 308 aninterface. Control programs for realizing the image processing functionsaccording to the present embodiment, and data used at the time of thecontrol programs being executed, are stored in the external storagedevice 304. The control programs and data are loaded to the RAM 302 viaa bus 309 as appropriate under control of the CPU 301, executed by theCPU 301, and function as the units described below. The functions of theblocks making up the information processing apparatus 10 will becorrelated with specific execution procedures of the informationprocessing apparatus 10 illustrated in the flowchart in FIG. 5.

Step S510: Image Acquisition

The image acquiring unit 110 requests the SLO imaging apparatus 20 toacquire wide-angle images D1 (D1 c, D1 r, D1 l) as illustrated in FIG.61, and high-magnification images (confocal images Dcj and non-confocalimages Dnrk and Dnik) at a rectangular region in the macular area, asindicated by Pr1 in FIG. 61. Acquisition of attribute data and fixationtarget positions F1 and Fcn, corresponding to these images, is alsorequested. Attribute information in the present embodiment is date ofimage acquisition, position of acquisition, focus position, image type(confocal/R channel/L channel/optional inter-image-type computation),number of gradations, field angle, resolution, and number of frames. Inresponse to this acquisition request, the SLO imaging apparatus 20acquires the wide-angle images D1 c, D1 r, and D1 l, confocal imagesDcj, and non-confocal images Dnrk and Dnik, and corresponding fixationtarget positions F1 and Fcn, and transmits these to data acquiring unit110. The data acquiring unit 110 receives the wide-angle images D1 c, D1r, and D1 l, confocal images Dcj, non-confocal images Dnrk and Dnik, andattribute data and fixation target positions F1 and Fcn, from the SLOimaging apparatus 20 via the LAN 30, and stores the received data in thestorage unit 120. Note that the acquisition position forhigh-magnification images is not restricted to the rectangular region inthe macular area, and images of optional acquisition positions may beused. For example, a case where an image is used that has been acquiredin a ring-shaped form around the optic disc as denoted by Pr2 in FIG. 61is also included in the present invention.

Step S520: Generating Composite Images

The determining unit 1311 determines which imaged images to use togenerate a composite image, based on the attribute data of the imagesacquired at step S510. Next, the image processing method deciding unit1312 decides the image generating method (type of computation betweenimages or within images, and number of gradations, resolution, andnumber of frames of the image to be generated) using the image forgeneration. The image processing unit 130 generates an image byperforming computation within or among confocal images (D1 n or Dn) bythe images for generating and the image generating method decided by thedetermining unit 1311 and image processing method deciding unit 1312.Further, the positioning unit 132 performs image positioning, and thedisplay control unit 133 displays confocal images and non-confocalimages. Specifics of the image generating processing will be describedlater in detail in S710, S720, S730, and S740 of FIG. 7A.

Step S530: Deciding Measurement Method

The determining unit 1311 determines images for measurement out of theimaged and generated images, based on the attributed data of the imagedimages (imaging position and image type), and the image processingmethod deciding unit 1312 decides a measurement method (type of imageprocessing, range of image processing, and interval of image processing)for the images to be measured. Specifics of the measurement methoddeciding processing will be described later in detail in S711 and S721.

Step S540: Measurement

The image processing unit 130 performs measurement processing based onthe image measurement method decided in S530, and the display controlunit 133 displays the measurement results. In the present embodiment,the image processing unit 130 performs detection and distributionmeasurement processing of photoreceptors, and the display control unit133 displays statistics such as photoreceptor density, along withphotoreceptor positions and a Voronoi diagram. Specifics of themeasurement processing will be described later in detail in S712, S722,S732, S742, and S752.

Step S550: Deciding Whether or not to Save Results

The instruction acquiring unit 140 externally acquires an instructionregarding whether or not to save in the data server 40 the imagesgenerated in S520, and the data of measurement results from S540, i.e.,statistics such as photoreceptor positions, Voronoi diagram,photoreceptor density, and so forth, regarding the confocal images Dcjand non-confocal images Dnsk. This instruction is input by an operatorby way of the keyboard 306 or mouse 307, for example. In a case wheresaving has been instructed (YES in S550) the flow advances to S560, andin a case where saving has not been instructed (NO in S550) the flowadvances to S570.

Step 560: Saving Results

The image processing unit 130 correlates the examination date andinformation identifying the examinee eye with the images decided to besaved in S550 and data related to the measurement results, and transmitsthis to the data server 40.

Step 570: Decision of Whether or not to End

The instruction acquiring unit 140 externally acquires an instructionregarding whether or not to end processing of the wide-angle images D1,high-magnification confocal images Dcj, and high-magnificationnon-confocal images Dnk, by the information processing apparatus 10.This instruction is input by an operator by way of the keyboard 306 ormouse 307, for example. In a case where an instruction for ending ofprocessing is acquired (YES in S570), the processing ends. On the otherhand, in a case of acquiring an instruction to continue processing (NOin S570), the flow returns to S510, and processing on the next examineeeye (or redoing the processing on the same examinee eye) is performed.The processing executed in S520 will be described in detail withreference to the flowchart in FIG. 7A.

Step S710: Acquiring Attribute Information of Imaged Image

The image processing unit 130 acquires attribute data of imaged images.Attribute information acquired in the present embodiment is date ofimage acquisition, position of acquisition, focus position, image type(confocal/R channel/L channel/optional inter-image-type computation suchas Split Detector image), resolution, number of gradations, and numberof frames.

Step S720: Deciding Image Generating Method

The determining unit 1311 determines which imaged images to use togenerate an image (image for generation (acquisition position and imagetype)), based on the attribute data of the imaged images. Next, theimage processing method deciding unit 1312 decides the image generatingmethod (type of computation between images or within images, and numberof gradations, resolution, and number of frames of the image to begenerated) using the image for generation. The image processing unit 130generates an image by performing computation within or amongnon-confocal images (D1 n or Dn) by the images for generating and theimage generating method decided by the determining unit 1311 and imageprocessing method deciding unit 1312. Further, the positioning unit 132performs image positioning, and the display control unit 133 displaysconfocal images and non-confocal images.

Specifically, in the present embodiment, the determining unit 1311performs determination regarding images for generating already existingat the time of imaging the photoreceptors, and confocal images Dcjnon-confocal images Dnk (R channel images Dnrk and L channel images Dn1k) taken within 1.5 mm from the fovea, the image processing methoddeciding unit 1312 makes a decision using the image for generating, sothat a composited image of confocal images Dcj and a composited image ofSplit Detector images are generated at the same number of gradients andthe same resolution as the original image. Accordingly, images of whichthe center of photography is farther than 1.5 mm from the fovea, and Rchannel images Dnrk and L channel images Dn1 k within 1.5 mm from thefovea are not used in image generation or measurement. Note that a SplitDetector image is a type of a differential image using non-confocalimages generated by computing ((pixel value of L-channel image−pixelvalue of R-channel image)/(pixel value of R-channel image+pixel value ofL-channel image)). The determination of the image for generating, andthe deciding of the image generating method, performed at the imageprocessing unit 130, are not restricted to the above. Any image forgenerating and image generating method may be used as long as an imagegenerating method whereby detailed observation and measurement can beperformed with a greater number of images for images that are crucialfor observation and measurement.

Arrangements where the image for generating or the image generatingmethod is decided based on the image attributes acquired via theinstruction acquiring unit 140 are also included in the presentinvention. FIGS. 6G and 6H illustrate examples of a confocal image Dcand Split Detector image Dns in a case of imaging photoreceptors P.

Step S730: Generating Images

The image processing unit 130 generates an image based on the image forgenerating that has been determined in S720, and the image generatingmethod that has been decided. Before generating an image, thepositioning unit 132 performs inter-frame positioning of wide-angleimage D1 c and confocal image Dc, and applies positioning parametervalues between frames to the wide-angle images D1 r and D1 l, andnon-confocal images Dnr and Dn1 as well. The Split Detector images D1 nsand Dns are generated by computation processing of already-positioned D1r, Dnr, and D1 l, Dn1, so there is no need to perform inter-framepositioning again. Specific frame positioning methods include thefollowing.

i) The positioning unit 132 sets a reference frame to serve as areference for positioning. In the present embodiment, the frame with thesmallest frame No. is the reference frame. Note that the frame settingmethod is not restricted to this, and any setting method may be used.

ii) The positioning unit 132 performs general inter-frame correlation(rough positioning). Although any positioning technique may be used, thepresent embodiment performs rough positioning using a correlationcoefficient as an inter-image similarity evaluation function, and Affinetransform as a coordinate conversion technique.

iii) The positioning unit 132 performs fine positioning based on thecorrespondence relation in the general position among the frames.

In the present embodiment, images that have been subjected to the roughpositioning obtained in ii) are then subjected to inter-frame finepositioning, using free form deformation (FFD), which is a type of anon-rigid positioning technique. Note that the fine positioningtechnique is not restricted to this, and any positioning technique maybe used. After image generation, the positioning unit 132 positionswide-angle image D1 c and high-magnification image Dcj, and finds therelative position of Dcj on D1 c. The positioning unit 132 acquires thefixation target position Fcn used at the time of imaging thehigh-magnification confocal images Dcj from the storage unit 120, to useas the initial point for searching for positioning parameters for thepositioning of the wide-angle image D1 c and confocal image Dcj. Thepositioning of the wide-angle image D1 c and high-magnification confocalimage Dcj is performed while changing combinations of the parametervalues. The combination of parameter values where the similarity betweenthe wide-angle image D1 c and high-magnification confocal image Dcj ishighest is decided to be the relative position of the confocal image Dcjas to the wide-angle image D1 c. Note that the positioning technique isnot restricted to this, and any positioning technique may be used.

Also, in a case where a mid-magnification image has been acquired inS510, positioning is performed from images with lower magnification. Forexample, in a case where a high-magnification confocal image Dc1 m and amid-magnification image Dc2 o have been acquired, first, the wide-angleimage D1 c and the mid-magnification image Dc2 o are positioned, andnext, the mid-magnification image Dc2 o and the high-magnificationconfocal image Dc1 m are positioned.

Further, image tiling parameter values decided regarding the wide-angleconfocal image D1 c and confocal image Dcj are applied to tiling of thenon-confocal images D1 r and Dnrk, D1 l and Dn1 k, D1 ns and Dnsk) aswell. The relative positions of the high-magnification non-confocalimages Dnrk, Dn1 k, and Dnsk on the wide-angle non-confocal images D1 r,D1 l, and D1 ns are each decided.

Step 740: Display

The display control unit 133 displays the generated image group on themonitor 305. An image obtained by compositing the confocal image Dcj andSplit Detector image Dnsk determined to be images for generating in S720(images of which the center of photography is within 1.5 mm from thefovea) are tiled here using the positioning parameter values decided inS730, and displayed. The type of images to be display is switched usinga graphical user interface (GUI) that has been prepared for thispurpose. Although radio buttons are used for switching in the presentembodiment, any GUI arrangement may be used for the switching. The typesof images to be switched are the two types of confocal image Dc andSplit Detector image Dns.

Now, the processing executed in S530 will be described in detail withreference to the flowchart of FIG. 7B.

Step S711: Acquiring Attribute Information of Imaged Image

The image processing unit 130 references the attribute data in theimaged images acquired in S710.

Step S721: Decide Measurement Method

The determining unit 1311 determines images to measure (acquisitionposition and image type), based on the attribute data of the imagedimages acquired in S711. Next, the image processing method deciding unit1312 decides the measurement method (type of image processing, range ofimage processing, intervals of image processing) regarding the images tomeasure.

The determining unit 1311 determines confocal images Dcj and SplitDetector images Dnsk of which the center of photography is within 1.5 mmfrom the fovea to be the present measurement object at the time ofimaging photoreceptors. That is to say, images of which the center ofphotography is farther than 1.5 mm from the fovea, and R channel imagesDnrk and L channel images Dn1 k within 1.5 mm from the fovea are not theobject of measurement. The image processing method deciding unit 1312decides to perform the following image processing on the compositedimage of confocal images Dcj and Split Detector images Dnsk of which thecenter of photography is within 1.5 mm from the fovea.

-   -   Detection of photoreceptor position    -   Creating a Voronoi diagram    -   Measuring photoreceptor density        There is the need to perform measurement on two types of images        and compare the measurement results under the same conditions in        the present embodiment, so the image processing range and image        processing interval (intervals between measurement positions)        are the same range, the same image size, and same distance for        both images as the original image. The present invention is not        restricted to this, and any image processing range and image        processing interval values may be set to the images, as long as        the processing is capable of measuring images crucial for        measurement in a more detailed manner.

Further, the processing executed in S540 will be described in detailwith reference to the flowchart in FIG. 7C.

Step S712: Filtering in Frequency Region

The image processing unit 130 removes high-frequency components, toremove peak components other than the photoreceptors in the confocalimages Dcj and non-confocal images Dnsk (noise and reflected light fromfundus tissue other than photoreceptors). The present embodimentperforms frequency conversion using fast Fourier transform (FFT), andcuts out high-frequency component signal value by applying a low-passfilter. The filtered images are restored to space domain by inverseFourier transform, thereby generating corrected confocal images andcorrected non-confocal images with the high-frequency component removed.

Step S722: Binarization

The photoreceptors P are detected by binarizing the corrected confocalimages and corrected non-confocal images using threshold values Tc andTs, respectively.

Step S732: Generating Voronoi Diagram

A Voronoi diagram is generated for the binarized image of thephotoreceptors P, following the procedures below. The center point ofeach photoreceptor region (MP in FIG. 6M) is calculated for thebinarized image of photoreceptors P, and a perpendicular bisector isdrawn for each line segment connecting a center point MP with anadjacent center point MP. The drawn perpendicular bisectors from themedian point of the line segments connecting the center points MP up tointersections VP with other perpendicular bisectors are retained and allother bisectors are erased, thereby obtaining Voronoi boundaries VB. Theareas and shapes that the individual photoreceptors P have arerepresented by regions VR enclosed by the Voronoi boundaries VB.

Step S742: Calculating Statistics of Photoreceptor Distribution

Statistics relating to the distribution of photoreceptors P arecalculated based in the Voronoi diagram generated in S732. Specifically,the density of the detected photoreceptors P, the average distancebetween adjacent photoreceptors, the average value of area that eachphotoreceptor occupies, the rate of region where photoreceptors P areexpressed in the form of hexagons in the Voronoi diagram, arecalculated. Note that the Statistics are calculated not only for theoverall image but also for each small region.

Step S752: Display

The display control unit 133 displays on the monitor 305:

i) corrected confocal images and corrected non-confocal images

ii) detection results of photoreceptors P

iii) Voronoi diagram

iv) map relating to statistics on photoreceptor distribution for eachsmall region.

The Voronoi diagram is displayed with each Voronoi region colored inaccordance with area. Note that the present embodiment has beendescribed with regard to an arrangement where the deciding unit 131decides methods for inter-image computation, image feature extraction(photoreceptor detection, blood vessel wall boundary detection, etc.),and image measurement, uniformly by methods specified for each imagingposition and image type, as necessary for observation and measurement,but the present invention is not restricted to this. For example,arrangements where the determining unit 1311 in the deciding unit 131uniformly determines which images are to be read into the storage unit120, and which images are to be displayed on the monitor 305, by methodsspecified for each imaging position and image type, as necessary forobservation and measurement, are also included in the present invention.Alternatively, the image processing method deciding unit 1312 mayuniformly decide methods to display images (resolution, number ofgradients, number of frames (including deciding whether moving image orstill image), etc.), by methods specified for each imaging position andimage type, as necessary for observation and measurement.

According to the configuration described above, the informationprocessing apparatus 10 uniformly performs image computation andcalculation uniformly by methods specified for each image attribute(imaging position and image type), as necessary for observation andmeasurement, on images where photoreceptors have been imaged using theSLO apparatus that acquires confocal images and non-confocal images atthe same time. Accordingly, images of the eye that are crucial forobservation and measurement can be efficiently generated or measured.

Although analyzing methods (example of image processing methods) aredecided beforehand for each of multiple types of images, a configurationmay be made where the analyzing method can be changed for each of themultiple types of images in accordance with user specification (e.g.,selecting an image processing method). Different analyzing methods arepreferably selected for each of the multiple image types. For example,confocal images may be given priority over non-confocal images, and atthis time, an analyzing method may be decided that the non-confocalimages are not analyzed. Conversely, non-confocal images may be givenpriority over confocal images, and at this time, an analyzing method maybe decided that the confocal images are not analyzed. Further, thedecided analyzing methods preferably include a method where the imagesare not analyzed. That is to say, deciding the analyzing methodpreferably includes deciding whether or not to analyze images. These arealso applicable to the following embodiments.

Second Embodiment Deciding Image Processing Method by Feature Regions inImage

An information processing apparatus according to a second embodimentperforms the following processing on images taken with an SLO apparatusthat acquires confocal images and non-confocal images at generally thesame time. The information processing apparatus is configured so that,based on image features or disorder candidate regions, the more cruciala portion for observation or image analysis included in an image is, thewider the variety or greater the detail of images that are generated is,or a wider variety of images are measured. Specifically, an SLOapparatus such as illustrated in FIGS. 3A and 3B that acquires confocalimages Dcj and non-confocal images Dnk at generally the same time,acquires confocal images Dcj and non-confocal images Dnk of retinalblood vessels at acquisition positions such as illustrated in FIG. 6J.Although a region where an initial disorder has damaged the outersegment but the inner segment has survived, this is imaged as a blackdefect area in confocal images (Dc5 in FIG. 6K), but can be observed asa region where high-luminance granular objects exist in non-confocalimages (Dn5 in FIG. 6L). A case will be described where the morearteriovenous crossing portions, which are areas of predilection forretinal vein occlusion, are included in an image, the wider the varietyor greater the detail of images that are generated is, or a widervariety of images are measured.

FIG. 2B illustrates the configuration of an apparatus connected to theinformation processing apparatus 10 according to the present embodiment.The present embodiment differs from the first embodiment in that theinformation processing apparatus 10 is connected to a time phase dataacquisition apparatus 50, in addition to the SLO imaging apparatus 20.The time phase data acquisition apparatus 50 is an apparatus thatacquires biosignal data (time phase data) that autonomously andcyclically changes, such as a sphygmograph or electrocardiograph, forexample. The time phase data acquisition apparatus 50 acquires timephase data Sj at the same time as acquiring high-magnification imagesDnk, in accordance with operations performed by an unshown operator. Theacquired time phase data Sj is sent to the information processingapparatus 10 and data server 40. Note that the time phase dataacquisition apparatus 50 may be directly connected to the SLO imagingapparatus 20.

In addition to the wide-angle images D1 r and D1 l andhigh-magnification images Dnrk and Dn1 k of the examinee eye, andacquisition conditions such as the fixation target positions F1 and Fcnused at the time of acquisition, the data server 40 also holds imagefeatures of the eye. Any image features of the eye may be used, but thepresent embodiment handles retinal blood vessels and capillaries Q, andphotoreceptor damage regions. The time phase data Sj output from thetime phase data acquisition apparatus 50 and image features output fromthe information processing apparatus 10 are saved in the server. Thetime phase data Sj and image features of the eye are transmitted to theinformation processing apparatus 10 upon request by the informationprocessing apparatus 10.

Next, FIG. 8 illustrates a functional block diagram of the informationprocessing apparatus 10 according to the present embodiment. Thisconfiguration differs from the first embodiment with regard to the pointthat the data acquiring unit 110 has a time phase acquiring unit 114,and the deciding unit 131 has an image feature acquiring unit 1313. Theimage processing flow according to the present embodiment is the same asthat illustrated in FIG. 5. Other than S510, S520, S530, and S540, theflow is the same as in the first embodiment, so just the processing ofS510, S520, S530, and S540 will be described in the present embodiment.

Step S510: Image Acquisition

The data acquiring unit 110 acquires wide-angle images D1 c and D1 n,confocal images Dcj, non-confocal images Dnk, and time phase data.Confocal images Dcj and non-confocal images Dnk are acquired following aretinal artery arcade A in the present embodiment, as illustrated inFIG. 6J. The time phase acquiring unit 114 requests the time phase dataacquisition apparatus 50 for time phase data Sj relating to biologicalsignals. In the present embodiment, a sphygmograph serves as a timephase data acquisition apparatus, used to acquire pulse wave data Sjfrom the earlobe of the subject. This pulse wave data Sj is expressedwith acquisition time on one axis and a cyclic point sequence having thepulse wave signal values measured by the sphygmograph on the other axis.The time phase data acquisition apparatus 50 acquires and transmits thetime phase data Sj corresponding to the acquisition request. The timephase acquiring unit 114 receives this pulse wave data Sj from the timephase data acquisition apparatus 50 via the LAN 30. The time phaseacquiring unit 114 stores the received time phase data Sj in the storageunit 120.

Now, there are two conceivable timings relating acquisition of the timephase data Sj by the time phase data acquisition apparatus 50; one is acase where the confocal data acquiring unit 111 or image processingmethod deciding unit 1312 starts image acquisition in conjunction with aparticular phase of the time phase data Sj, the other is a case whereacquisition of pulse wave data Pi and image acquisition aresimultaneously started immediately after an image acquisition request.In the present embodiment, acquisition of pulse wave data Pi and imageacquisition are simultaneously started immediately after an imageacquisition request. The time phase data Pi of each image is acquired bythe time phase acquiring unit 114, the extreme value in each time phasedata Pi is detected, and the cardiac cycle and relative cardiac cycleare calculated. The relative cardiac cycle is a relative valuerepresented by a floating-point number between 0 and 1 where the cardiaccycle is 1. The data acquiring unit 110 requests the SLO imagingapparatus 20 for acquisition of wide-angle images D1 c, D1 r, and D1 l,confocal images Dcj, non-confocal images Dnrk and Dn1 k, andcorresponding fixation target position F1 and Fcn data.

In response to the request, the SLO imaging apparatus 20 acquires andtransmits the wide-angle images D1 c, D1 r, and D1 l, confocal imagesDcj, non-confocal images Dnrk and Dn1 k, and corresponding fixationtarget positions F1 and Fcn. The data acquiring unit 110 receives thewide-angle images D1 c, D1 r, and D1 l, confocal images Dcj,non-confocal images Dnrk and Dn1 k, and corresponding fixation targetpositions F1 and Fcn, from the SLO imaging apparatus 20 via the LAN 30,and stores these in the storage unit 120. FIGS. 6C and 6D illustrate anexample of a confocal image Dc and non-confocal image Dnr in a case ofhaving photographed a retinal blood vessel. The confocal image Dc hasstrong reflection from the nerve fiber layer, so the background noisemakes positioning difficult. The non-confocal image Dnr (Dn1) of the Rchannel (L channel) has higher contrast at the blood vessel wall at theright side (left side).

On the other hand, examples of non-confocal images are not limited tothis. Other examples include an addition process image Dnr+1 ofnon-confocal images Dnr and Dn1, and a Split Detector images Dns towhich a type of differential processing ((L−R)/(R+L)) has been applied.FIGS. 6E and 6F illustrate examples of Dnr+1 and Dns. “Non-confocalimage Dnk” below can refer to any of these non-confocal images Dnk.

Step S520: Generating Images

The image feature acquiring unit 1313 acquires retinal blood vesselregions and arteriovenous crossing portions as image features from thewide-angle images D1 n. Although the image features are acquired in S520in the present embodiment, acquiring the image features in this step isnot restrictive. For example, cases where the image features areacquired immediately after image acquisition in S510, and cases wherethe image features are acquired in S530, are also included in thepresent invention. Next, determination is made regarding which imagedimage (image for generating) to use to generate an image is determinedby the determining unit 1311 based on the image features. Further, theimage processing method deciding unit 1312 determines the imagegenerating method (type of inter-image or intra-image computation, andnumber of gradients, resolution, and number of frames of generatedimage) based on the image features. Further, the image processing unit130 performs computation among non-confocal data (D1 n or Dn) accordingto the computation contents decided by the image processing methoddeciding unit 1312 to generate images, and the positioning unit 132executes image positioning. The display control unit 133 displaysconfocal images and non-confocal images. Specific image generatingprocessing will be described in detail in S910, S920, S930, and S940.

Step S530: Deciding Measurement Method

The determining unit 1311 determines images to be measured (imagingposition and image type) from the images that have been imaged andgenerated, based on the image features (retinal blood vessel regions andarteriovenous crossing portions) acquired by the image feature acquiringunit 1313 in S520. Further, the image processing method deciding unit1312 decides the measurement method for the images to be measured (typeof image processing, range of image processing, and interval of imageprocessing) for the images to be measured, based on the image features.Specifics of the measurement method deciding processing will bedescribed later in detail in S911 and S921.

Step S540: Measurement

The image processing unit 130 performs measurement processing based onthe image measurement method decided in S530, and the display controlunit 133 displays the measurement results. In the present embodiment,the image processing unit 130 detects retinal blood vessel walls andperforms wall thickness measurement processing, and the display controlunit 133 displays detected wall boundaries, wall thickness graphs, andwall thickness maps. Specifics of the measurement processing will bedescribed later in detail in S912, S922, S932, and S942.

Next, the processing executed in S520 will be described in detail withreference to the flowchart in FIG. 9A.

Step S910: Image Feature Acquisition

The image feature acquiring unit 1313 detects the retinal blood vesselregions and arteriovenous crossing portions as image features from thewide-angle images D1 n. The images from which image features areacquired are not restricted to wide-angle images, and cases of directlyacquiring image features from high-magnification images Dnk, forexample, are also included in the present invention. Retinal bloodvessels have linear shapes, so the present embodiment employs a retinalblood vessel region detection method where a filter that enhances linearstructures is used to extract the retinal blood vessels. Specifically, awide-angle image D1 n is smoothed by a Gaussian function of a size aequivalent to the radius of the arcade blood vessel, and thereupon atube enhancement filter based on a Hessian matrix is applied, andbinarization is performed at a threshold value Ta, thus extracting thearcade blood vessels.

As for the method for detecting arteriovenous crossing portions in thepresent embodiment, a crossing detection filter disclosed in JapanesePatent Laid-Open No. 2001-070247 is used. Specifically, a crossingportion is determined when there are four or more blood vessel regionsat the perimeter of the filter, and there is a blood vessel region atthe center portion of the filter. Retinal arteries contain morehemoglobin than retinal veins and thus are higher in luminance, so thelowest luminance value within each of the crossing blood vessel regionsis calculated from the detected crossing portions, and in a case wherethe absolute value among the lowest luminance values is equal to orlarger than a threshold value T1, this is determined to be anarteriovenous crossing. Note however, that the crossing detection methodis not restricted to this, and any known crossing detection method maybe used. Retinal veins tend to be lower with regard to luminancedistribution of the intravascular space region (the region where bloodflows), so in the present invention, if the lowest luminance value inthe intravascular space region in confocal images and non-confocalimages is smaller than Tv, this is identified as a retinal vein in thepresent invention, and if Tv or larger, as a retinal artery.

Although description has been made in the present embodiment that theimage feature acquiring unit 1313 acquires anatomical features such asarteriovenous crossing portions, the present invention is not restrictedto this. For example, a disorder candidate region such as thephotoreceptor defect portion Dc5 in FIG. 6K may be acquired as an imagefeature. Although any detection method of the photoreceptor defectregion may be used, detection is made in the present embodimentaccording to the following procedures. That is to say, Fourier transformis performed in the confocal images Dcj, a low-pass filter is applied tocut out high-frequency signal values, following which inverse Fouriertransform is performed, and each region having a value smaller than athreshold T2 is detected as a photoreceptor defect region.

Step S920: Deciding Image Generating Method

The determining unit 1311 determines which imaged image (image forgenerating) to use to generate an image, based on the image featuresacquired by the image feature acquiring unit 1313 in S910. Further, theimage processing method deciding unit 1312 decides the image generatingmethod (type of computation between images or within images, and numberof gradations, resolution, and number of frames of the image to begenerated), based on the image features. In retinal vein occlusion,which is a common eye disorder, arteriovenous crossing portions are anarea of predilection (blockage of retinal veins), so in the presentembodiment, images including arteriovenous crossing portions are used togenerate images with large total data amounts (many types of images orimages with large data amount). Specifically, the determining unit 1311determines an image including a retinal blood vessel area to be an imagefor generating. The image processing method deciding unit 1312 decidesthe image type for generating to be a (R+L) image Dnr+1. The imageprocessing method deciding unit 1312 further decides to generate, withregard to an image including an arteriovenous crossing portion, a movingimage having the same resolution as the original image in 16-bit and acomposited image having the same resolution as the original image in16-bit, and with regard to an image not including an arteriovenouscrossing portion, a composited image having the same resolution as theoriginal image in 16-bit. However, this is not restrictive, and anyimage type to be generated and any generating method thereof may bespecified, as long as an image generating method that enables a greaternumber of images to be observed and measured in detail with regard toimages that are crucial for observation and measurement. For example,the image processing method deciding unit 1312 may decide the image typeto be generated to be (R+L) images Dnr+1 and Split Detector images Dnsfor images including arteriovenous crossing portions, and just (R+L)images Dnr+1 for images not including arteriovenous crossing portions.

In a case of acquiring a disorder region (disorder candidate) which isan example of a state that is the object of observation, such asphotoreceptors of the eye, as image features from a confocal image Dc instep S910, the image for generating is decided according to thefollowing procedures. A photoreceptor defect region, which is an exampleof a disorder region, will be described here. It is generally understoodthat photoreceptors, which are an example of observation, becomedefective from the outer segments, next become defective at the innersegments, and finally reach necrosis. Scoles and Dubra describe thatconfocal images Dcj enable photoreceptor outer segment defects to beobserved, while non-confocal images Dnk enable photoreceptor innersegment defects to be observed. Accordingly, it can be understood thatphotoreceptor defect regions in confocal images (at least imagesincluding photoreceptor outer segment defects) and Split Detector imagesat the same imaging position, are important for observation and analysisof the level of the photoreceptor disorder. Thus, the determining unit1311 determines, from the imaged images, confocal images at all imagingpositions, and non-confocal images Dn at the same imaging positions asconfocal images containing photoreceptor defect regions in the confocalimages, to be images for generating (imaging position and image type).Further, the image processing method deciding unit 1312 further decidesto generate a composited image having the same resolution as theoriginal image in 16-bit, for all confocal images Dcj, regardless ofwhether or not a photoreceptor detect region is included, and a SplitDetector image Dnsk using a generating R channel image Dnrk and Lchannel image Dn1 k in 16-bit and as a composited image having the sameresolution as the original image in 16-bit. However, this is notrestrictive, and any image type to be generated and any generatingmethod thereof may be specified, as long as an image generating methodthat enables a greater number of images to be observed and measured indetail with regard to images that are crucial for observation andmeasurement.

Step S930: Generating Images

The image processing unit 130 generates images by performing computationamong non-confocal images (D1 n or Dn) according to the computationdecided by the image processing method deciding unit 1312, and thepositioning unit 132 executes image positioning. In the presentembodiment, the image processing unit 130 generates (R+L) images (D1nr+1 and Dnr+1). Computation by the image processing unit 130 is notrestricted to this, and any computation processing may be performed. Thepositioning unit 132 performs inter-frame positioning and compositing ofgenerated wide-angle images D1 r+1 and non-confocal images Dnr+1, andfurther performs tiling processing of composited wide-angle images D1r+1 and composited high-magnification images Dn(r+1)k. Moreover, thedisplay control unit 133 displays the generated image group on themonitor 305. Specific inter-frame positioning and tiling procedures arethe same as in the first embodiment, and accordingly description will beomitted here. The inter-frame positioning parameters decided here arealso applied to the wide-angle images D1 r, D1 l, and D1 c, andhigh-magnification images Dnr, Dn1, and Dc, that have the same imagingposition. In the same way, the image tiling parameter values decidedwith regard to the wide-angle images D1 r+1 and non-confocal imagesDnr+1 are applied to tiling of confocal images (D1 c and Dcj) and tilingof non-confocal images (D1 r, Dnrk, and Dn1 k) as well.

On the other hand, with regard to measurement of blood vessel wallthickness, composited images are generated according to the followingprocedures to avoid change in the shape of the blood vessels amongframes due to influence of the heartbeat. That is to say, instead ofaveraging all frames, just frames correlated with pulse wave signalsbelonging to a particular phase interval are selected and composited. Inthe present embodiment, the phase intervals of pulse waves are dividedinto five intervals, and the frames belonging to the interval includingthe phase where the pulse wave signal value is minimal are selected andcomposited. However, the frame selection method is not restricted tothis, and any selection method can be used yields the effects ofeliminating the influence of the heartbeat can be used.

Step S940: Display

The display control unit 133 displays the generated images. In thiscase, the images generated in S930 are tiled using the positioningparameters described above, and displayed.

The processing performed in S530 will be described in detail withreference to the flowchart illustrated in FIG. 9B.

Step S911: Acquiring Image Features

The image processing method deciding unit 1312 references the imagefeatures acquired by the image feature acquiring unit 1313 in S910(retinal blood vessel regions and arteriovenous crossing portions, orthe photoreceptor defect portion Dc5).

Step S921: Deciding Measurement Method

The determining unit 1311 performs determination (imaging position andimage type) of the images that are objects of measurement, based onimage features acquired in S911. Next, the image processing methoddeciding unit 1312 decides the image measurement method (type of imageprocessing, range of image processing and intervals of image processing)for the images that are objects of measurement. In retinal veinocclusion, which is a common eye disorder, arteriovenous crossingportions are an area of predilection (blockage of retinal veins), so inthe present embodiment, detailed measurement processing is performedregarding images including arteriovenous crossing portions.

Specifically, the determining unit 1311 determines images including aretinal blood vessel area to be images for measuring. Next, the imageprocessing method deciding unit 1312 decides

-   -   Edge preservation smoothing processing    -   Blood vessel wall boundary detection    -   Blood vessel wall thickness measurement        to be types of image processing to be performed on the (R+L)        images Dr+1 at imaging positions including arteriovenous        crossing portions, out of the measurement methods for the images        to be measured. With regard to the range of image processing,        edge preservation smoothing is performed for the entire image,        and blood vessel wall boundary detection and blood vessel wall        thickness measurement are decided only for arterial branches        crossing veins. As for images that do not include arteriovenous        crossing portions, just edge preservation smoothing processing        is applied as a type of image processing to the entire image        every two pixels, and measurement processing is not applied.        However, this is not restrictive, and any determining processing        for the images to be measured and deciding processing for the        image processing method may be specified, as long as an image        processing method that enables a greater number of images to be        measured or measured in detail with regard to images having        image features that are crucial for measurement. Note that in a        case of acquiring disorder candidates such as a photoreceptor        defect region from the confocal images Dcj and non-confocal        images Dnk as image features in S911, the measurement method is        decided according to the following procedures.

In the same way as in S920, a Split Detector image having the sameimaging position as a photoreceptor defect region in a confocal imagecan be though to be a crucial image in observing and analyzing thedegree of the photoreceptor disorder. Accordingly, the determining unit1311 determines confocal images containing a photoreceptor defect regionand Split Detector images having the same imaging position as theconfocal images, out of the images that have been imaged and generated,to be images to be measured (imaging position and image type). That isto say, confocal images Dc that do not contain a photoreceptor defectregion are determined not to be the object of measurement. Further,based on the image features, the image processing method deciding unit1312 decides, as the image measurement method (type of image processing,range of image processing, and interval of image processing) for theimages that are objects of measurement,

-   -   Detection of photoreceptor position    -   Creating a Voronoi diagram    -   Measuring photoreceptor density        so that all images a processed over the same range as the        original image, in one-pixel intervals. However, this is not        restrictive, and any type of image processing, range of image        processing, and interval of image processing may be specified        for each image, as long as processing that enables a wider        variety of images to be measured or measured in detail with        regard to images having image features that are crucial for        measurement.

Next, the processing performed in S540 will be described in detail withreference to the flowchart illustrated in FIG. 9C.

Step S22: Smoothing

The image processing unit 130 performs smoothing processing on the imageto be measured. Although any known edge preservation smoothingprocessing may be applied, in the present embodiment a median valuefilter is applied in one-pixel intervals to the entire image ofcomposited images of (R+L) images including arteriovenous crossingportions. On the other hand, the median filter is applied in two-pixelintervals to the entire image of images containing other blood vesselregions. This edge preservation smoothing processing enables noisewithin the image to be reduced without blurring the edges of bloodvessel wall boundaries.

Step S922: Detecting Wall Boundaries

The image processing unit 130 detects boundary positions of retinalarteries. First, a Top-hat filter, which is a type of morphology filter,is applied to the smoothed image generated in S912. By applying theTop-hat filter, a slender high-luminance region that is present near thecenter of a blood vessel (blood column reflection) is extracted, andthereafter the extracted region is formed into a fine line, therebydetecting the center line of the artery. Note that “Top-hat filterprocessing” refers to is processing where luminance values of an openingimage (an image where the original image is subjected to reductionprocessing and then to expansion processing) are subtracted from theluminance values of the original image. The Top-hat filter may beapplied to an image obtained by the original image having beenbinarized, or a multi-value Top-hat filter may be applied to theoriginal image and then binarization performed. The method for detectingthe center line of the blood vessel is not restricted to this, and anyknown method may be used.

Next, luminance profiles are generated on line segments that passthrough control points set equidistantly on the center line and aregenerally perpendicular to the center line. Two maximum values each areextracted for the left side of the line segment from the center and forthe right side of the line segment from the center, and these are takenas the blood vessel wall boundaries. Note that the blood vessel wallboundary detection method is not restricted to this, and any knowntechnique may be used. For example, the following technique may be used.Four cures parallel to the center line of the blood vessel aregenerated, with two curves disposed on the intravascular space region,and two are disposed on the outside of the blood vessel, as a deformablemodel. Minimizing an evaluation function stipulated regarding shapesbetween control points making up the model and luminesce values on thecontrol points results in the four deformable models being deformed tomatch the blood vessel wall boundaries, thereby detecting the bloodvessel wall boundary positions.

Step S932: Measuring Wall Thickness

The image processing unit 130 calculates the outer diameter of the bloodvessel, the inner diameter of the blood vessel, the wall thickness, andWall-to-Lumen Ratio (WLR) and Wall Cross-Sectional Area (WCSA) followingalong arterial branches that are the object of measurement, as indexvalues based on the wall thickness, based on the blood vessel boundarypositions detected in S922, where the following hold.

WLR=(outer diameter of blood vessel−inner diameter of bloodvessel)/(inner diameter of blood vessel)

WCSA=π((outer diameter of blood vessel)²−(inner diameter of bloodvessel)²)

Step S942: Display

The display control unit 133 displays a graph on the monitor 305 of theouter diameter of the blood vessel, the inner diameter of the bloodvessel, the wall thickness, and wall thickness index values, measuredfollowing along the blood vessel. That is to say, the position in thedirection in which the blood vessel runs is displayed on the horizontalaxis, and the outer diameter of the blood vessel, the inner diameter ofthe blood vessel, the wall thicknesses on the left and right sides, andwall thickness index values (WLR and WCSA) are displayed on the verticalaxis. Note however, that the display method of measurement values andindex values is not restricted to this, and that the type of measurementvalue or index value may be selected and the display switched.Displaying this sort of wall thickness graph facilitates usercomprehension of wall thickness distribution along the blood vessel. Thedisplay control unit 133 also performs a superimposed display of thewall boundaries detected in S922 following along the blood vessel, andthe measurement values and wall thickness index values measured in S932,upon a confocal image or non-confocal image, or a composited imagethereof. The measurement values and wall thickness index values may bedisplayed in grayscale or may be displayed as a color map correlatedwith an optional color bar. Displaying such a distribution of wallthicknesses as a map facilitates user comprehension of wall thicknessvalues at the fundus.

Although description has been in the present embodiment made regarding acase where the deciding unit 131 performs inter-image computation andextracts image features (photoreceptor detection, blood vessel wallboundary detection, etc.) based on image features and disorder candidateregions, and decides the method regarding image measurement the presentinvention is not restricted to this. For example, an arrangement wherethe determining unit 1311 of the deciding unit 131 determines whichimage to read into the storage unit 120 or which image to display on themonitor 305, based on image features and disorder candidate regions, isalso included in the present invention. Further, the image processingmethod deciding unit 1312 may decide the image display method(resolution, number of gradients, number of frames (including whether amoving image or still image), etc.) based on image features and disordercandidate regions.

According to the above-described configuration, The informationprocessing apparatus 10 performs the following processing on imagestaken by an SLO apparatus to acquires confocal images and non-confocalimages at generally the same time. Based on image features and disordercandidate regions acquired in the images, the more crucial a portion forobservation or image analysis included in an image is, the wider thevariety or greater the detail of images that are generated is, or awider variety of images are measured. Accordingly, eye images that arecrucial for observation and analysis can be efficiently generated ormeasured.

Third Embodiment Deciding Image Processing Method by Image AnalysisResults

An information processing apparatus according to a third embodiment isconfigured to decide an image generating method or measuring methodbased on not only image attributes or image features (disordercandidates) acquired from the image, but also analysis results fromanalyzing the image, which is to say image quality, and percentage of animaging object region in an image. Specifically, description will bemade regarding an image generating method or measuring method in a casewhere the same retinal blood vessels are taken as confocal images Dcjand two types of non-confocal images Dnk (non-confocal images taken withthe aperture opened to a large diameter and moved to the right side andleft side along the retinal blood vessels). The configuration ofapparatuses connected to the information processing apparatus 10according to the present embodiment is the same as in the case in thesecond embodiment. Note however, than the reception method of confocalsignals and non-confocal signals by the SLO imaging apparatus 20 in thepresent embodiment differs, in that a configuration is made to where thediameter and position of the aperture (pinhole) is variable, so as to beable to receive as confocal signals as in FIG. 3C or to be able toenlarge the diameter of aperture as in FIG. 3D and to move the apertureso as to receive non-confocal signals.

FIG. 10 is a functional block diagram of the information processingapparatus 10 according to the present embodiment, which differs from thearrangement in the second embodiment with regard to the point that thedeciding unit 131 includes a conformability calculating unit 1314. Theimage processing flow according to the present embodiment is the same asin FIG. 5, and is the same as the second embodiment except for S510,S520, and S530. Accordingly, S510, S520, and S530 will be described inthe present embodiment.

Step S510: Acquiring Images

The image acquiring unit 110 requests the SLO image imaging apparatus 20and time phase data acquisition apparatus 50 to acquire wide-angleimages Dc1 and Dn1, confocal image Dcj, and two types of non-confocalimages (Dnr and Dn1) at the imaging position as in the secondembodiment, and corresponding attribute data and fixation targetpositions F1 and Fcn, and time phase data. Attribute data in the presentembodiment is date of image acquisition, position of acquisition, focusposition, image type (confocal/R channel/L channel/optionalinter-image-type computation), number of gradations, field angle,resolution, and number of frames. In response to this acquisitionrequest, the SLO imaging apparatus 20 and time phase data acquisitionapparatus 50 acquire the wide-angle images D1 c, D1 r, and D1 l,confocal images Dcj, and non-confocal images Dnrk and Dn1 k, andcorresponding attribute data and fixation target positions F1 and Fcn,and time phase data Sj, and transmits these. The data acquiring unit 110receives the wide-angle images D1 c, D1 r, and D1 l, confocal imagesDcj, and non-confocal images Dnrk and Dn1 k, and corresponding attributedata and fixation target positions F1 and Fcn, and time phase data Sj,from the SLO imaging apparatus 20 and time phase data acquisitionapparatus 50 via the LAN 30, and stores these in the storage unit 120.Hereinafter, of the two types of non-confocal images, images acquired bymoving the aperture to the right side of the retinal blood vessels willbe denoted by Dnr, and images acquired by moving the aperture to theleft side of the retinal blood vessels will be denoted by Dn1. Theseimages are acquired with the aperture (pinhole) opened wide.

Step S520: Generating Images

The image feature acquiring unit 1313 acquires the retinal blood vesselregions and arteriovenous crossing portions as image features from thewide-angle images D1 n. Further, the conformability calculating unit1314 calculates the conformability based on the image quality ofacquired image, and the percentage that the region actually imagedoccupies in the region to be imaged, and the determining unit 1311 andimage processing method deciding unit 1312 determine the imagegenerating method based on the image features and conformability. Theimage processing unit 130 generates images based on the decision made bythe image processing method deciding unit 1312. Further, the positioningunit 132 performs image positioning, and the display control unit 133displays confocal images and non-confocal images. Specific imagegenerating processing will be described in detail in S1110 through S1150of FIG. 11A. Note that an arrangement may be made where the attributeacquiring unit 113 acquires the attribute data rather than the imagefeature acquiring unit 1313 acquiring image features in this step.

Step S530: Deciding Measurement Method

The image processing method deciding unit 1312 decides the measurementmethod based on the image features (or attribute data) andconformability of the image acquired in S520. Specific measurementmethod deciding processing will be described in detail in S1111, S1121,and S1131.

Next, the processing executed in S520 will be described in detail withreference to the flowchart illustrated in FIG. 11A.

Step S1110: Acquiring Attribute Information and Image Features

The image feature acquiring unit 1313 acquires retinal blood vesselregions and arteriovenous crossing portions as image features from thewide-angle images D1 n. The method of acquiring the retinal blood vesselregions and arteriovenous crossing portions is the same as in the secondembodiment, so description will be omitted. Cases where the imagefeature acquiring unit 1313 acquires photoreceptor defect regions fromthe confocal image Dcj as image features as in step S910 in the secondembodiment are also included in the present invention. The specificmethod of acquiring photoreceptor defect region is the same as in thesecond embodiment, so description will be omitted. Also, an arrangementmay be made where the attribute acquiring unit 113 acquires attributedata in this step, instead of the image feature acquiring unit 1313acquiring image features. Attribute data in the present embodiment isdate of image acquisition, position of acquisition, focus position,image type (confocal/R channel/L channel/optional inter-image-typecomputation), number of gradations, field angle, resolution, and numberof frames.

Step S1120: Calculating Image Conformability

The conformability calculating unit 1314 calculates the conformabilitybased on the image quality of each image, and the percentage of theregion actually imaged occupies in the region to be imaged. For imagequality, the signal-to-noise (S/N) ratio is calculated. The index of theimage quality is not restricted to this, and any known index may beused. For example, Contrast-Noise Ratio (CNR) may be calculated. Also, avalue indicating whether the region to be imaged has been sufficientlyimaged is calculated by (area of region imaged in all frames)/(area ofregion to be imaged). In the present embodiment, conformability isexpressed as ω1·Iq+ω2·Ic, where Iq is 1 if the S/N ratio is equal to orlarger than a threshold T3, and is 0 if the S/N ratio is smaller thanthreshold T3. Ic is (area of region imaged in all frames)/(area ofregion to be imaged). ω1 and ω2 are weighting parameters that can bespecified optional values in the range of 0 to 1. Both are 0.5 in thepresent embodiment.

Step S1130: Deciding Image Generating Method

The determining unit 1311 determines which imaged image (image forgenerating) to use to generate an image, based on the image featuresacquired by the image feature acquiring unit 1313 in S1110 andconformability calculated by the conformability calculating unit 1314 inS1120. Further, the image processing method deciding unit 1312 decidesthe image generating method (type of computation between images, andnumber of gradations, resolution, and number of frames of the image tobe generated), based on the image features and conformability. Thedetermining unit 1311 determines an R channel image Dnr and L channelimage Dn1, including a retinal blood vessel region, that have aconformability or threshold T4 or higher, to be the object ofgenerating. The image processing method deciding unit 1312 generates,with regard to images Dnr and Dn1 including an arteriovenous crossingportion out of the images for generating, a 16-bit moving image of (R+L)image Dnr+1 having the same resolution as the original image, and acomposited image having the same resolution as the original image in16-bit, and on the other hand, with regard to images Dnr and Dn1 notincluding an arteriovenous crossing portion, an 8-bit composited imageof (R+L) image Dnr+1 having the same resolution as the original image in8-bit.

In a case of the image feature acquiring unit 1313 acquiring aphotoreceptor defect region as image features, as in S910 in the secondembodiment, the following processing is performed. The determining unit1311 determines all confocal images, and non-confocal images Dnk at thesame imaging positions as confocal images Dcj containing photoreceptordefect regions, to be images for generating (imaging position and imagetype). Further, the image processing method deciding unit 1312 decidesto generate a composited image having the same resolution as theoriginal image in 16-bit using the confocal images included in theimages for generating. The image processing method deciding unit 1312also decides to generate a Split Detector image using a generating Rchannel image and L channel image included in the images for generating,in 16-bit as a composited image having the same resolution as theoriginal image in 16-bit. However, this is not restrictive, and anyimage type to be generated and any generating method thereof may bespecified, as long as an image generating method that enables a greaternumber of images to be observed and measured, or measure in detail, withregard to images that are crucial for observation and measurement.

Step S1140: Generating Images

The image processing unit 130 generates an image by performingcomputation among confocal images (D1 n or Dn) using the images forgenerating determined by the determining unit 1311 in S1130 and theimage generating method decided by the image processing method decidingunit 1312. Further, the positioning unit 132 performs image positioning.Specific image generating procedures that the image processing unit 130performs in this step are the same as in the second embodiment, exceptfor the point that the image for generating determined based onconformability (or the decided image generating method) differs, sodetailed description will be omitted.

Next, the processing performed in S530 will be described in detail withreference to the flowchart illustrated in FIG. 11B.

Step S1111: Acquiring Attribute Information and Image Features

The image processing unit 130 references the attribute data acquired bythe attribute acquiring unit 113 or the image features acquired by theimage feature acquiring unit 1313 in S1110.

Step S1121: Calculating Image Conformability

The image processing unit 130 references the conformability calculatedby the conformability calculating unit 1314 in S1120.

Step S1131: Deciding Measurement Method

The determining unit 1311 performs determination (imaging position andimage type) of the images that are objects of measurement, based onimage features acquired in S1111. Next, the image processing methoddeciding unit 1312 decides the image generating method (type of imageprocessing, range of image processing and intervals of image processing)for the images that are objects of measurement.

In the present embodiment, the number of images for generating havealready been narrowed down (Dnr and Dn1 that have conformability ofthreshold T4 or higher determined to be images for generating) based onconformability in the image generating method deciding processing(S1130), so the objects of measurement are not narrowed down furtherbased on conformability in this step. Accordingly, the image processingmethod deciding unit 1312 decides

-   -   Edge preservation smoothing processing    -   Blood vessel wall boundary detection    -   Blood vessel wall thickness measurement        to be types of image processing to be performed on the (R+L)        images Dr+1 at imaging positions including arteriovenous        crossing portions, having conformability of threshold T4 or        higher. With regard to the range of image processing, edge        preservation smoothing is performed for the entire image, and        blood vessel wall boundary detection and blood vessel wall        thickness measurement are decided only for arterial branches        crossing veins. The image processing interval is decided to be        in 1-pixel increments. Note that an arrangement may be made        where no narrowing down of the images for generating is        performed based on conformability at the time of deciding the        image generating method in S1130, and instead the images that        are the object of measurement may be narrowed down in this step        based on conformability (along with the condition of images        having conformability of threshold T4 or higher). Further, a        configuration may be made where different conformability        conditions may be set in the image generating method deciding        processing in S1130 and this step, and measurement is performed        on images satisfying both.

While the conformability has been described as being calculated by theconformability calculating unit 1314 based on the image quality and thepercentage including the region to be imaged, the calculating method ofconformability is not restricted to this, and any calculating method maybe used as long as based on coordinates and pixel values of the imagedimages. For example, the conformability may be calculated based onluminance characteristics of the imaged images, i.e., luminance valuesand statistics relating to the luminance values. Specifically, theaverage luminance value of the image that has been imaged may becalculated, and a value where this average luminance value is weightedmay be used as the conformability. Alternatively, the contrast of theimaged images (maximum luminance−minimum luminance)/(maximumluminance+minimum luminance) may be calculated as conformability.

Although an arrangement has been described in the above embodiment wherethe deciding unit 131 performs inter-image computation and extractsfeatures (photoreceptor detection, blood vessel wall boundary detection,etc.) based on image attributes or image features and image acquisitionresults (image quality, etc.), and decides a method relating to imagemeasurement, the present invention is not restricted to this. Forexample, arrangements where the determining unit 1311 in the decidingunit 131 determines which image to read into the storage unit 120 orwhich image to display on the monitor 305, based on image attributes orimage features and image acquisition results (image quality, etc.), arealso included in the present invention. Alternatively, the imageprocessing method deciding unit 1312 may decide methods to displayimages (resolution, number of gradients, number of frames (includingdeciding whether moving image or still image), etc.), based on imageattributes or image features and image acquisition results (imagequality, etc.).

According to the configuration described above, the informationprocessing apparatus 10 decides generating methods and measuring methodsfor image using not only image attributes or image features (disordercandidates) acquired from the images, but also using image acquisitionresults, i.e., image quality and percentage including region to beimaged. Thus, images of the eye that are crucial for observation andmeasurement can be efficiently generated or measured.

Fourth Embodiment Deciding Image Processing Method Based on ProcessingResults by Examination Date

The information processing apparatus 10 according to the presentembodiment is configured such that, based on images acquired ondifferent examination dates, the more examination dates including areference examination date that generating and measurement have beenperformed there are in attributes of an image, the easier to is for thatimage to be determined to be an image for generating or an image formeasurement. Specifically, in a case where confocal images andcorresponding non-confocal images including a photoreceptor outersegment defect region have been acquired on different examination dates,the attributes of these image groups taken on different examinationdates are acquired, the following processing is performed. That is tosay, description will be made regarding a case where the moreexamination dates including a reference examination date that generatingand measurement have been performed there are in attributes of an image,the easier to is for that image to be determined to be an image forgenerating or an image for measurement. The configuration of apparatusesconnected to the information processing apparatus 10 according to thepresent embodiment is as illustrated in FIG. 2A, and differs from thethird embodiment with regard to the point no time phase data acquisitionapparatus is connected. The functional block configuration of theinformation processing apparatus 10 according to the present embodimentdiffers from that in the third embodiment with regard to the points thatthe data acquiring unit 110 does not have the time phase acquiring unit114, the data acquiring unit 110 acquires examination data of differentexamination dates, the deciding unit 131 does not have the image featureacquiring unit 1313, and the conformability calculating unit 1314calculates conformability as to base examination images based on theattributes of images on different examination dates.

The image processing flow according to the present embodiment isillustrated in FIG. 5, and is the same as the third embodiment exceptfor S510, S520, and S530. Accordingly, S510, S520, and S530 will bedescribed in the present embodiment.

Step S510: Acquiring Images

The data acquiring unit 110 requests the data server 40 to transfer pastconfocal images Dcjf (where f=1, 2, . . . , e−1, f being a naturalnumber indicating the No. of the examination in serial order), pastnon-confocal images Dnkf, and fixation target positions Fcnf. The dataserver 40 transfers the data corresponding to the request to theinformation processing apparatus 10, and the data is saved in thestorage unit 120. In the present embodiment, e=5, and Split Detectorimages Dnsf are acquired as non-confocal images Dnkf of pastexaminations.

Next, the data acquiring unit 110 requests the SLO imaging apparatus 20for acquisition of wide-angle images D1 ce and D1 ne, confocal imagesDcje and non-confocal images Dnke, and fixation target positions F1 eand Fcne corresponding to the newest examination. In the presentembodiment, the wide-angle images D1 ce and D1 ne, confocal images Dcje,and non-confocal images Dnke are acquired with fixation target positionsF1 e at the fovea and fixation target positions Fcne at the fovea andparafovea. Note that the imaging position setting method is notrestricted to this, and the imaging position may be set to an optionalposition. The SLO imaging apparatus 20 acquires and transmits thewide-angle images D1 ce and D1 ne, confocal images Dcje and non-confocalimages Dnke, and fixation target positions F1 e and Fcne, in response tothe acquisition request. The data acquiring unit 110 receives thewide-angle images D1 ce and D1 ne, confocal images Dcje, non-confocalimages Dnke, and fixation target positions F1 e and Fcne, from the SLOimaging apparatus 20 via the LAN 30. The data acquiring unit 110 storesthe received wide-angle images D1 ce and D1 ne, confocal images Dcje,non-confocal images Dnke, and fixation target positions F1 e and Fcne inthe storage unit 120.

Step S520: Generating Images

Upon the attribute acquiring unit 113 having acquired attribute data forall examination images, and the conformability calculating unit 1314having calculating the conformability with past examination images, thedetermining unit 1311 and image processing method deciding unit 1312decide images for generating and image generating method, based on theconformability. The image processing unit 130 generates images based onthe decision of the determining unit 1311 and image processing methoddeciding unit 1312. Further, the positioning unit 132 performspositioning, and the display control unit 133 displays confocal imagesand non-confocal images. Specifics of the image generating processingwill be described in detail in S1112, S1122, S1132, S1142, and S1152.

Step S530: Deciding Measurement Method

Upon the conformability calculating unit 1314 having calculated theconformability of the images of the current examination with the pastexamination images based on attribute data of the past examinationimages, the determining unit 1311 and image processing method decidingunit 1312 decide images for measuring and measuring method, based on theconformability. Specifics of the measurement method deciding processingwill be described in detail in S1113, S1123, and S1132.

Next, the processing performed in S520 will be described in detail withreference to the flowchart illustrated in FIG. 11C.

Step 1112: Acquiring Attribute Information of all Examination Images

The attribute acquiring unit 113 acquires attribute data relating to thepast examination images and the current examination images. Attributedata in the present embodiment is date of image acquisition, position ofacquisition, focus position, image type (confocal/R channel/Lchannel/optional inter-image-type computation), number of gradations,field angle, resolution, and number of frames.

Step 1122: Calculating Inter-Examination Conformability

The conformability calculating unit 1314 calculates conformability withpast examination images for each image, based on the attribute data ofpast examination images. In the present embodiment, conformability ofthe current examination images as to the past examination images iscalculated by ω3·Ib+ω4·ΣIen for each of the current examination images.Ib here is a value (0 or 1) representing whether or not there is animage having the same attributes other than the date of acquisition in areference examination (baseline). Ien is calculated by (value (0 or 1)representing whether or not there is an image having the same attributesother than the date of acquisition)/(total number of past examinations).Σ represents the summation of all past examinations. ω3 and ω4 areweighting parameters that can be given optional values in the range of 0to 1. Both are 0.5 in the present embodiment.

Step 1132: Deciding Image Generating Method

The determining unit 1311 determines an image for generating based onthe conformability with past examination images calculated in S1122, andthe image processing method deciding unit 1312 decides an imagegenerating method. In the present embodiment, the determining unit 1311determines that images of which the conformability calculated in S1122is equal to or larger than a threshold T5 use confocal images Dcj andSplit Detector images Dnsk as images for generating, and images of whichthe conformability calculated in S1122 is smaller than threshold T5 useonly confocal images Dcj as images for generating. Also, the imageprocessing method deciding unit 1312 decides to generate the former as16-bit moving images and composited images, and the later as 16-bitcomposited images having half the resolution of the original image. Thedetermination of images for generating and image generating methodsbased on the calculated conformability is not restricted to theprocedures described here. Any determination of images for generatingand image generating method determination may be performed as long asthe more examination dates including a reference examination date thatgenerating and measurement have been performed there are in attributesof an image, the easier to is for that image to be determined to be animage for generating or an image for measurement.

Step 1142: Generating Images

The image processing unit 130 generates an image based on the decisionthat the image processing method deciding unit 1312 has made in S1132.Next, the positioning unit 132 performs inter-frame positioning on thewide-angle images D1 cf (where f=1, 2, . . . , e) for each examinationdata, and confocal images Dcjf. The specific procedures for positioningframes are the same as in S520 in the first embodiment, so descriptionwill be omitted. Next, the positioning unit 132 performs inter-framepositioning of the wide-angle images D1 nf and non-confocal images Dbkf,using the positioning parameter values decided for the wide-angle imagesD1 cf and confocal images Dcjf. The positioning unit 132 performspositioning among the wide-angle images D1 cf (where f=1, 2, . . . , e)for each examination data, and the confocal images Dcjf, and calculatesthe relative position of the confocal images Dcjf upon the wide-angleimages D1 cf. In a case where there is an overlapping region amongconfocal images D1 cf, first, the inter-image similarity is calculatedrelating to the overlapping region, and the confocal images Dcjf arepositioned at the position where the inter-image similarity is thegreatest. In a case where images of three or more different types ofmagnification have been acquired in S510, positioning is performed inorder from lower-magnification images. Any known technique may be usedfor the inter-image similarity and coordinate conversion techniques.Positioning is performed in the present embodiment using correlationcoefficients for inter-image similarity, and Affine transformation asthe coordinate transform technique. A tiled image of confocal imagesDcjf is generated using the information of relative positions of theconfocal images Dcjf on the wide-angle images D1 cf obtained by thispositioning processing. Next, the positioning unit 132 generates a tiledimage of non-confocal images Dnkf using the information of relativepositions of the confocal images Dcjf on the wide-angle images D1 cf.

Further, positioning is performed between reference wide-angle imagesand wide-angle images D1 cf and D1 nf other than the referencewide-angle images. The reference wide-angle images, confocal images, andnon-confocal images, can be selected from any examination date of imagesacquired in S510. In the present embodiment, an image group of theoldest examination date (wide-angle images D1 c 1 and D1 n 1, confocalimages Dcj1, and non-confocal images Dnk1) are each taken as referenceimages.

The relative position of the reference image D1 c 1 as to the referenceimage Dcj1, the relative position of other examination images as to thereference image D1 c 1, and the relative position of the confocal imagesDcjf as to the images D1 cf, are used to find the relative position ofthe confocal images Dcjf as to the reference image Dcj1. Note that thereference images Dcj1 and Dcjf may be directly positioned. Any knowntechnique may be used for positioning. Positioning is performed in thepresent embodiment using Affine transformation as primary generalpositioning. Next, free form deformation (FFD), which is a type of anon-rigid positioning technique, is used to perform detailedpositioning. In either positioning, correlation coefficients are usedfor inter-image similarity. Of course, this is not restrictive, and anyknown image similarity technique may be used. Next, the positioning unit132 uses information the relative position of the confocal images Dcjfas to the reference image Dcj1 to find the relative position of thenon-confocal images Dnkf as to the reference image Dck1.

Thus, the pixels of the reference images (D1 c 1, D1 n 1, Dcj1, andDnk1) and the images (D1 cf, D1 nf, Dcjf, and Dnkf) other than thereference images are correlated. Note that the present invention is notrestricted to positioning based on similarity of pixel values, and anarrangement may be made where blood vessel regions are identified, uponwhich positioning is performed based on features using the identifiedblood vessel regions.

Step 1152: Display

The display control unit 133 displays the image group generated so faron the monitor 305. Here, the composited images are displayed tiledusing the above-described positioning parameters. Also, compositedimages, and in a case where moving images have been generated,inter-frame-positioned moving images, are displayed regarding a imagingposition instructed via the instruction acquiring unit 140.

Next, the processing performed in S530 will be described in detail withreference to the flowchart illustrated in FIG. 11D.

Step 1113: Acquiring Attribute Information of All Images

The image processing unit 130 references the attribute data of allexamination images acquired by the attribute acquiring unit 113 inS1112.

Step 1123: Calculating Inter-Examination Conformability

The image processing unit 130 references the conformability calculatedby the conformability calculating unit 1314 in S1122.

Step 1133: Deciding Measurement Method

The determining unit 1311 determines images to be measured (imagingposition and image type) based on the inter-examination conformabilityreferenced in S1123, and the image processing method deciding unit 1312decides the image generating method (type of image processing, range ofimage processing, and interval of image processing). Further, there isthe need for the image processing method deciding unit 1312 to performmeasurement on two types of images and compare the measurement resultsregarding

-   -   Detection of photoreceptor position    -   Creating a Voronoi diagram    -   Measuring photoreceptor density        under the same conditions, so both images are processed over the        same measurement range as the original image, in one-pixel        intervals. However, this is not restrictive, and any type of        image processing, range of image processing, and interval of        image processing may be measured for each image, as long as        processing that enables more detailed measurement with regard to        images that are crucial for measurement.

Although description has been made in the above embodiment where thedeciding unit 131 decides methods relating to inter-image computationand image feature extraction (photoreceptor detection, blood vessel wallboundary detection, etc.), and image measurement, based on the degree ofhaving attributes where more examination dates on which generating andmeasurement have been performed, the present invention is not restrictedto this. For example, arrangements where the determining unit 1311 inthe deciding unit 131 determines which image to read into the storageunit 120 or which image to display on the monitor 305, based on thedegree of having attributes where more examination dates on whichgenerating and measurement have been performed, are also included in thepresent invention. Alternatively, the image processing method decidingunit 1312 may decide methods to display images (resolution, number ofgradients, number of frames (including deciding whether moving image orstill image), etc.), based on the degree of having attributes where moreexamination dates on which generating and measurement have beenperformed.

According to the information processing apparatus 10 configured asdescribed above, based on images acquired on different examinationdates, the more examination dates including a reference examination datethat generating and measurement have been performed there are inattributes of an image, the easier to is for that image to be determinedto be an image for generating or an image for measurement. Accordingly,in a case of comparing and observing images of different examinationdates, eye images that are crucial for observation and analysis can beefficiently generated or measured.

OTHER EMBODIMENTS

Although description has been made that the data acquiring unit 110includes both the confocal data acquiring unit 111 and the non-confocaldata acquiring unit 112, the data acquiring unit 110 does not need toinclude the non-confocal data acquiring unit 112, as long as theconfiguration enables acquisition of two or more types of non-confocaldata.

OTHER EMBODIMENTS

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions recorded on a storage medium (e.g., non-transitorycomputer-readable storage medium) to perform the functions of one ormore of the above-described embodiment(s) of the present invention, andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or more ofa central processing unit (CPU), micro processing unit (MPU), or othercircuitry, and may include a network of separate computers or separatecomputer processors. The computer executable instructions may beprovided to the computer, for example, from a network or the storagemedium. The storage medium may include, for example, one or more of ahard disk, a random-access memory (RAM), a read only memory (ROM), astorage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-093540, filed Apr. 30, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information processing apparatus comprising:an image acquiring unit configured to acquire a plurality of types ofimages of an eye, including a confocal image and a non-confocal image ofthe eye; an analyzing unit configured to analyze the confocal image andthe non-confocal image; a deciding unit configured to decide, based onanalysis results of one of the confocal image and the non-confocalimage, whether or not to analyze the other; and a display control unitconfigured to display analysis results of the confocal image and thenon-confocal image on a display unit, in a case of deciding to analyzethe other.
 2. The information processing apparatus according to claim 1,wherein the analyzing unit determines a state of an object ofobservation in the confocal image by analyzing the confocal image, andwherein the deciding unit decides whether or not to analyze thenon-confocal image based on the state of the object of observation. 3.The information processing apparatus according to claim 2, wherein thestate of the object of observation is whether or not there is a disorderregion, and in a case where there is a disorder region, the decidingunit decides to analyze the non-confocal image.
 4. The informationprocessing apparatus according to claim 3, wherein the state of theobject of observation is whether or not there is a photoreceptor defectregion, and in a case where there is a photoreceptor defect region, thedeciding unit decides to analyze the non-confocal image.
 5. Aninformation processing apparatus comprising: an image acquiring unitconfigured to acquire a plurality of types of images of an eye,including a confocal image and a non-confocal image of the eye; adeciding unit configured to decide, based a state of an object ofobservation of one of the confocal image and the non-confocal image,whether or not to generate the other of the confocal image and thenon-confocal image; and a display control unit configured to display thegenerated other on a display unit, in a case of deciding to generate theother.
 6. The information processing apparatus according to claim 5,wherein the state of the object of observation is whether or not thereis a disorder region, and in a case where there is a disorder region,the deciding unit decides to generate the other.
 7. The informationprocessing apparatus according to claim 6, wherein the state of theobject of observation is whether or not there is a photoreceptor defectregion, and in a case where there is a photoreceptor defect region, thedeciding unit decides to generate the other.
 8. An informationprocessing apparatus comprising: an image acquiring unit configured toacquire a plurality of types of images of an eye, including anon-confocal image of the eye; a deciding unit configured to decide ananalysis method to analyze the plurality of types of images that havebeen acquired; and an analyzing unit configured to analyze at least oneof the plurality of types of images that have been acquired, based onthe analysis method that has been decided.
 9. The information processingapparatus according to claim 8, wherein the deciding unit decides theanalysis method based on at least one of the plurality of type of imagesthat have been acquired and an attribute of the plurality of type ofimages that have been acquired.
 10. The information processing apparatusaccording to claim 8, wherein the deciding unit decides the analysismethod based on at least one of anatomical features and disordercandidates, in the plurality of type of images that have been acquired.11. The information processing apparatus according to claim 8, whereinthe deciding unit decides the analysis method based on at least one ofimage quality, luminance characteristics, conformability to instructedimaging conditions, and conformability to an image attribute of a pastexamination, in the plurality of type of images that have been acquired.12. The information processing apparatus according to claim 8, whereinthe deciding unit decides at least one of image reading, inter-imagecomputation, image feature extraction, image measurement, and imagedisplay, as the analysis method.
 13. The information processingapparatus according to claim 8, wherein the deciding unit decideswhether or not to analyze the plurality of types of images that havebeen acquired, as the analysis method.
 14. The information processingapparatus according to claim 1, wherein the information processingapparatus is communicably connected to an ophthalmologic imagingapparatus that a plurality of types of images of the eye, and whereinthe image acquiring unit acquires a plurality of types of imagesobtained by imaging the eye at generally the same time.
 15. Theinformation processing apparatus according to claim 14, wherein theophthalmologic imaging apparatus includes a shared light source toacquire a confocal image and a non-confocal image of the eye, and anoptical member that splits returning light from the eye irradiated bylight form the light source, into returning light passing through aconfocal region and returning light passing through a non-confocalregion, and wherein the image acquiring unit acquires the confocal imagebased on the returning light passing through the confocal region, andacquires the non-confocal image based on the returning light passingthrough the non-confocal region.
 16. The information processingapparatus according to claim 15, wherein the image acquiring unitacquires the confocal image and non-confocal image of the eye, obtainedby adjusting at least one of a position and a shape of an aperturedisposed upstream of a light-receiving portion that receives light of atleast one of returning light passing through the confocal region andreturning light passing through the non-confocal region.
 17. Theinformation processing apparatus according to claim 15, wherein anacquisition position of the confocal image and an acquisition positionof the non-confocal image of the eye are the same.
 18. An operationmethod of an information processing apparatus, the method comprising:acquiring a plurality of types of images of an eye, including a confocalimage and a non-confocal image of the eye; deciding, based on analysisresults of one of the confocal image and the non-confocal image, whetheror not to analyze the other; and displaying analysis results of theconfocal image and the non-confocal image on a display unit, in a caseof deciding to analyze the other.
 19. An operation method of aninformation processing apparatus, the method comprising: acquiring aplurality of types of images of an eye, including a confocal image and anon-confocal image of the eye; deciding, based on a state of an objectof observation in one of the confocal image and the non-confocal image,whether or not to generate the other of the confocal image and thenon-confocal image; and displaying the generated other on a displayunit, in a case of deciding to analyze the other.
 20. An operationmethod of an information processing apparatus, the method comprising:acquiring a plurality of types of images of an eye, including anon-confocal image of the eye; deciding an analysis method to analyzethe plurality of types of images that have been acquired; and analyzingat least one of the plurality of types of images that have beenacquired, based on the analysis method that has been decided.